{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8650b1f1-0874-468c-8478-e7b72ad45c8b",
   "metadata": {},
   "source": [
    "## How to create a metric on top of a python function?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eec35259-4541-482f-b77d-7b2edadb29e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import pandas as pd\n",
    "import requests\n",
    "import zipfile\n",
    "\n",
    "from datetime import datetime\n",
    "from datetime import time\n",
    "from datetime import timedelta\n",
    "from dateutil.relativedelta import relativedelta\n",
    "from sklearn import ensemble\n",
    "from sklearn.metrics import explained_variance_score, r2_score\n",
    "\n",
    "from evidently import ColumnMapping\n",
    "\n",
    "from evidently.base_metric import InputData\n",
    "from evidently.metrics import RegressionQualityMetric\n",
    "from evidently.metrics.custom_metric import CustomValueMetric\n",
    "from evidently.report import Report\n",
    "from evidently.renderers.html_widgets import WidgetSize\n",
    "from evidently.test_suite import TestSuite\n",
    "from evidently.tests.custom_test import CustomValueTest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b91e1a74-e091-4bc9-9198-56afa60344d2",
   "metadata": {},
   "source": [
    "### Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2db8c29f-0acb-401a-a32e-f0ec0cbb9bfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "content = requests.get(\n",
    "        \"https://archive.ics.uci.edu/static/public/275/bike+sharing+dataset.zip\",\n",
    "        verify=False,\n",
    "    ).content\n",
    "\n",
    "with zipfile.ZipFile(io.BytesIO(content)) as arc:\n",
    "    raw_data = pd.read_csv(\n",
    "        arc.open(\"hour.csv\"),\n",
    "        header=0,\n",
    "        sep=\",\",\n",
    "        parse_dates=[\"dteday\"],\n",
    "        index_col=\"dteday\",\n",
    "    )\n",
    "\n",
    "raw_data.index = raw_data.apply(\n",
    "    lambda row: datetime.combine(row.name, time(hour=int(row[\"hr\"]))) + relativedelta(years=11),\n",
    "    axis=1,\n",
    ")\n",
    "raw_data.sort_index(inplace=True)\n",
    "\n",
    "reference = raw_data.loc[\"2023-01-01 00:00:00\":\"2023-01-28 23:00:00\"]\n",
    "current = raw_data.loc[\"2023-01-29 00:00:00\":\"2023-02-28 23:00:00\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "570bc268-ff4d-40e5-8a3c-5a9e57b26f01",
   "metadata": {},
   "source": [
    "### Define column mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2bb61d5-b3d3-43af-aa6e-f1828688dd2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "target = \"cnt\"\n",
    "prediction = \"prediction\"\n",
    "numerical_features = [\"temp\", \"atemp\", \"hum\", \"windspeed\", \"hr\", \"weekday\"]\n",
    "categorical_features = [\"season\", \"holiday\", \"workingday\"]\n",
    "\n",
    "column_mapping = ColumnMapping()\n",
    "column_mapping.target = target\n",
    "column_mapping.prediction = prediction\n",
    "column_mapping.numerical_features = numerical_features\n",
    "column_mapping.categorical_features = categorical_features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e85eb06-028c-4b2c-a9ac-77d3e664250f",
   "metadata": {},
   "source": [
    "### Train regression model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58ae4fbe-c870-49bf-a21f-3d31bcabb54e",
   "metadata": {},
   "outputs": [],
   "source": [
    "regressor = ensemble.RandomForestRegressor(random_state=0, n_estimators=50)\n",
    "regressor.fit(reference[numerical_features + categorical_features], reference[target])\n",
    "\n",
    "reference[\"prediction\"] = regressor.predict(reference[numerical_features + categorical_features])\n",
    "current[\"prediction\"] = regressor.predict(current[numerical_features + categorical_features])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "080e1f58-e2c9-47ba-b086-dcf67b52e221",
   "metadata": {},
   "source": [
    "### Implement functions that return a single value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "783b8a19-ddd8-4172-b9ba-1013a01db13d",
   "metadata": {},
   "source": [
    "We use ```InputData``` as a function argument.\n",
    "\n",
    "```InputData``` has the following structure:\n",
    "```\n",
    "    reference_data: Optional[pd.DataFrame]\n",
    "    current_data: pd.DataFrame\n",
    "    reference_additional_features: Optional[pd.DataFrame]\n",
    "    current_additional_features: Optional[pd.DataFrame]\n",
    "    column_mapping: ColumnMapping\n",
    "    data_definition: DataDefinition\n",
    "    additional_data: Dict[str, Any]\n",
    "```\n",
    "\n",
    "This basically means that you can access any attribute from ```InputData``` in your custom metric.\n",
    "\n",
    "for more details checkout the code here: https://github.com/evidentlyai/evidently/blob/main/src/evidently/base_metric.py "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecea23e0-3767-4fc6-9b20-07b5b65b507f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def variance_func(data: InputData): \n",
    "    return explained_variance_score(data.current_data[data.column_mapping.target],\n",
    "        data.current_data[data.column_mapping.prediction])\n",
    "\n",
    "def r2_func(data: InputData): \n",
    "    return r2_score(data.current_data[data.column_mapping.target],\n",
    "        data.current_data[data.column_mapping.prediction])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0388eba1-5f91-48bf-94d3-80b426c57f3f",
   "metadata": {},
   "source": [
    "In the functions above we used ```current_data``` and ```column_mapping``` from ```InputData```.\n",
    "\n",
    "For instance, if you need to acces ```reference_data```, ```additional_data``` or even ```data_definition``` you can access them all similarly."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "828b4a35-6647-4445-b08e-fecd68df2cbe",
   "metadata": {},
   "source": [
    "### Wrap functions in a CustomValueMetric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "969e6320-13ed-46a8-bdfb-ed8b53bae487",
   "metadata": {},
   "outputs": [],
   "source": [
    "report = Report(\n",
    "    metrics=[\n",
    "        RegressionQualityMetric(),\n",
    "        CustomValueMetric(func=r2_func, title=\"Current: R2 score\", size=WidgetSize.HALF),\n",
    "        CustomValueMetric(func=variance_func, title=\"Current: Variance\", size=WidgetSize.HALF),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e92cda66-5291-4be1-aaaa-0e4f5a0a5b70",
   "metadata": {},
   "outputs": [],
   "source": [
    "report.run(\n",
    "    reference_data=reference,\n",
    "    current_data=current,\n",
    "    column_mapping=column_mapping,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db93419e-a45f-4378-9c9f-88dac5892fd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "report.show(mode='inline')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abb857f3",
   "metadata": {},
   "source": [
    "### Wrap a function in a CustomValueTest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8edea52a",
   "metadata": {},
   "outputs": [],
   "source": [
    "suite = TestSuite(tests=[CustomValueTest(func=r2_func, title='Current R2 score', lte=0.9, gte=0.7)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6dcada9",
   "metadata": {},
   "outputs": [],
   "source": [
    "suite.run(\n",
    "    reference_data=reference,\n",
    "    current_data=current,\n",
    "    column_mapping=column_mapping\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d18d645a",
   "metadata": {},
   "outputs": [],
   "source": [
    "suite.show(mode='inline')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58b22b4b-7eda-42a1-a608-0cb7ee3dc2a3",
   "metadata": {},
   "source": [
    "## How to use a Custom Metric in a monitoring dashboard?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08ca47d8-61c7-4332-a42b-450fdde381c9",
   "metadata": {},
   "source": [
    "Import relevant evidently modules to work with monitoring:\n",
    "\n",
    "```{python}\n",
    "from evidently.ui.workspace.cloud import CloudWorkspace\n",
    "\n",
    "from evidently.ui.dashboards import CounterAgg\n",
    "from evidently.ui.dashboards import DashboardPanelCounter\n",
    "from evidently.ui.dashboards import PanelValue\n",
    "from evidently.ui.dashboards import ReportFilter\n",
    "from evidently.renderers.html_widgets import WidgetSize\n",
    "```\n",
    "\n",
    "Set up your workspace:\n",
    "```{python}\n",
    "workspace = CloudWorkspace(\n",
    "        token=\"YOUR_TOKEN\",\n",
    "        url=\"https://app.evidently.cloud/\"\n",
    "        )\n",
    "```\n",
    "\n",
    "Create or load a project:\n",
    "```{python}\n",
    "project = ws.create_project(\"PROJECT_TITLE\")\n",
    "```\n",
    "\n",
    "Create panels on top of custom metrics:\n",
    "```{python}\n",
    "project.dashboard.add_panel(\n",
    "        DashboardPanelCounter(\n",
    "            title=\"Custom Metric: R2\",\n",
    "            filter=ReportFilter(metadata_values={}, tag_values=[]),\n",
    "            value=PanelValue(\n",
    "                metric_id=\"CustomValueMetric\",\n",
    "                metric_args={\"title\": \"R2 score\"},\n",
    "                field_path=CustomValueMetric.fields.value,\n",
    "                legend=\"count\",\n",
    "            ),\n",
    "            text=\"count\",\n",
    "            agg=CounterAgg.LAST,\n",
    "            size=WidgetSize.HALF,\n",
    "        )\n",
    ")\n",
    "\n",
    "project.dashboard.add_panel(\n",
    "    DashboardPanelCounter(\n",
    "        title=\"Custom Metric: Variance\",\n",
    "        filter=ReportFilter(metadata_values={}, tag_values=[]),\n",
    "        value=PanelValue(\n",
    "            metric_id=\"CustomValueMetric\",\n",
    "            metric_args={\"title\": \"Variance\"},\n",
    "            field_path=CustomValueMetric.fields.value,\n",
    "            legend=\"count\",\n",
    "        ),\n",
    "        text=\"count\",\n",
    "        agg=CounterAgg.LAST,\n",
    "        size=WidgetSize.HALF,\n",
    "    )\n",
    ")\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
